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REMARKS — General 

By the above amendment, Applicant has amended the claims to define the invention more 
particularly and distinctly so as to overcome the technical rejections and define the invention 
patentably over the prior art. 
Claim 1 has been canceled. 

Claims 2, 3, 5, and 7 have been canceled. All of the limitations in claims 5 and 7 have been 
integrated into the newly written claim 22 as limitations of the newly made independent claim to 
define patentably over the reference. 

The limitations from 2(a) to 2(d) in claim 2 have been integrated into the newly written claim 22 
as limitations of the newly made independent claim. The limitation 2(e) in claim 2 has been 
integrated into the new claim 24 as a limitation of the new claim to more particularly define the 
invention in a patentable manner over the cited prior art. 

The limitations 3(a) and 3(b) in claim 3 have been integrated into the newly written claim 22 as 
limitations of the newly made independent claim. The limitations 3(c) and 3(d) in claim 3 have 
been rewritten as new claims 25 and 26 respectively to more particularly define the invention in 
a patentable manner over the cited prior art. 

Claims 4, 6, 9, 10, 11, and 12 have been rewritten as new claims 30, 28, 29, 27, 31, and 32, 
respectively, to more particularly define the invention in a patentable manner over the cited prior 
art. 

Claim 8 has been integrated into the newly written claim 24 as a limitation to more particularly 
define the invention in a patentable manner over the cited prior art. 
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Claims 13 and 17 have been canceled. The limitations in claims 13 and 17 have been integrated 
into the newly written claim 33 as limitations of the newly made independent claim to define 
patentably over the reference. 
Claims 14, 15, and 16 have been canceled. 

Claims 18, 19, 20, and 21 have been rewritten as new claims 35, 36, 42, and 41, respectively, to 
more particularly define the invention in a patentable manner over the cited prior art. 
The Rejection Of The Claims Under § 102 

The O.A. rejected claim 1 under 35 U.S.C. 102(b) as being anticipated by Gao et al. (6508553, 
referred to as "Gao" herein). 

According to the last O.A., claim 1 has been canceled. 
The Rejection Of The Claims Under § 103 

The O.A. rejected claims 2-21 under 35 U.S.C. 103(a) as being unpatentable over Gao et al. 

(6508553, referred to as "Gao" herein) in view of Soatto (6944327). 

Applicants amended the claims as follows: 

The Rejection Of Claim 2 on Gao and Soatto Overcome 

Regarding claim 2, the last O.A. noted that Gao discloses (a) initialization module further 
comprises steps for: (b) processing a single image or a plurality of images from said captured 
plurality of images in order to detect face for an initial face detection (column 13 line 15 to 
column 16 line 24 disclose detecting the facial edge and detecting the facial shape), (d) verifying 
said face (column 14 lines 8-29 disclose classifying the facial shape detection), and (e) 
estimating regions of interest for each facial features in said face (column 14 line 30 to column 
15 line 23 discloses finding regions of interest from the initial face detection). 
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The last O.A. also noted that Gao does not explicitly disclose steps (a) and (c), but Soatto 
discloses (a) capturing a plurality of images for an individual or a plurality of people with a 
single or a plurality of means for capturing images (column 10 lines 15-35 disclose receiving a 
sequence of images), (c) tracking said detected face in real-time (column 10 lines 24-35 disclose 
tracking a segmented face throughout a sequence of images). 

Claim 2 has been canceled and the limitations in claim 2 have been integrated into the newly 
written claim 22 as limitations of the newly made independent claim to define patentably over 
the reference. 

Applicants request reconsideration of the rejection, as now applicable to claim 22, for the 
following reasons: 

(1) Novel and unobvious approaches in applicants' system are clearly foreign to Gao and Soatto. 
Several differences in the fundamental approaches are discussed hereinafter. The significant 
differences in the approaches between applicants' system and Gao and Soatto or any 
combination thereof should not be underestimated or misunderstood. 

(2) There is no justification, in Gao and Soatto, or in any other prior art separate from applicants' 
disclosure, which suggests that these references be combined, much less be combined in the 
manner proposed. Since the fundamental approaches in applicants' system are clearly foreign to 
Gao and Soatto, even if Gao and Soatto were to be combined in the manner proposed in the last 
O.A., the proposed combination would not show all the novel features of claim 22. 

(3) The block processing in the application of the plurality of facial feature detection algorithms, 
such as the mean-crossing algorithms, and the integration design of each key process in 
applicants' system for real-time and automatic facial image enhancement in uncontrolled 
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background environment as proposed in applicants' disclosure, produce new and unexpected 

results and hence are unobvious and patentable over these references. 

The Reference and Differences of the Present Invention Thereof 

Prior to discussing the claim and the above three points, applicants will first discuss the 

references and the general novelty of the present invention and its unobviousness over the 

references. 

Overall, Gao and Soatto do not show the novel features in applicants' proposed system, which 
can comprise the following: 

(1) Gao and Soatto did not explicitly disclose a facial image enhancement by other types of 
virtual object images than eyewear. The eyewear or eyeglass frame selection system by Gao and 
Soatto is one of the many exemplary facial image enhancement systems. While Gao and Soatto 
only target the eyewear, applicants disclosed a novel facial image enhancement system that 
comprises not only eyewear superimposition, but also non-eyewear virtual object images, such as 
hat, cap, visor, or necklace, superimposition. 

(2) Gao and Soatto are clearly foreign to the novel usage of the two-level automatic face 
detection processes, in which a first global face detection process is used for an initial face 
detection and a second local face detection process is used for the verification of face within the 
boundary of the face tracking window, which enables a novel and efficient real-time face 
tracking. 

(3) Motion-based touch-free interaction is clearly foreign to Gao and Soatto, and it is a novel 
approach over any prior art. 
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(4) There are fundamental differences between applicants' system and Gao and Soatto or any 
combination thereof in processing the facial feature detection and application of the technology. 
Gao and Soatto did not disclose the novel approach in the incorporation of multiple real-time 
facial feature detection processing algorithms, such as incorporation of the mean-crossing 
algorithm and the color-based facial feature detection method in the context of real-time facial 
feature detection. In addition, the novel usage of block processing in applicants' system is clearly 
foreign to Gao and Soatto or any combination thereof 

(5) Gao required a non-automatic initialization step for the facial feature detection and Soatto 
required a cumbersome calibration step, which the applicants' system considers as obstacles to 
overcome. 

(6) Real-time face tracking, facial feature detection, and superimposition that overcome the 
issues and challenges in an uncontrolled real-world environment are clearly foreign to Gao and 
Soatto. 

(7) Gao assumed a fixed user position and a fixed size for the face and facial features, whereas 
applicants' approach handles the dynamic movement and changing size of the user position and 
facial features in real-time and automatically. 

(8) High resolution images were assumed in Gao, and handling low-resolution images for the 
facial feature detection and superimposition of the virtual object images in the manner proposed 
in applicants' system are foreign to Gao and Soatto. 

Non-Eyewear Facial Image Enhancement System 
Is Clearly Foreign To Gao and Soatto 
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Gao disclosed an interactive eyewear selection system, which is one of the many possible 
applications of the facial image enhancement system. Gao is foreign to the concept of other 
facial image enhancement systems for other predefined virtual objects, which can be attached to 
and enhance the human face image than the eyewear, such as a hat. 

Soatto disclosed a method and apparatus for designing and visualizing the shape of eyeglass 
lenses and of the front rims of eyeglass frames and for allowing the customer to modify the 
design by changing the shape and style interactively. Soatto did not explicitly disclose any other 
facial image enhancement, such as a hat image superimposition, than model visualization of the 
selected eyeglass frame that appears to move with the user's face. 

Whereas, the super-imposer in applicants' system "receives facial feature detection information 

501 about the face measurement from the facial feature detection module 201, and it initialize its 
internal variables with the information, such as eye position 608, left face-line 612, right face- 
line 613, vertical center line 614 of the face image 800, uppermost-line 606 of the face image 
800, and lowermost-line 607 of the face image 800. It sets the style of the virtual object images 

502 at the initial stage, and the super-imposer applies different measures depending on the style 
of the selected object image 503. For example, if the object style is set with regard to the eye 
positions, such as a pair of sunglasses, it uses the eye position information 608, and if it is related 
to the top of the face, such as a hat image 104, it uses the position information from the 
uppermost-line 606 of the face image 800." (Applicants, page 22, lines 1-11). 

In applicants, "the image of the user may be superimposed with a pair of sunglasses image 103, 
hat image 104, or any other predefined virtual objects, which can be attached to and enhance the 
human face image 800." (Applicants, page 7, lines 9-12). Gao is clearly foreign to the "hat image 
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104" in applicants' FIG. 1 and "a headwear 600, such as hats or visor" in applicants' FIG. 6. 
Thus, Gao does not explicitly discuss about the difficulties in setting pivot points in the virtual 
objects for non-eyewear. Whereas, applicants explicitly discussed "the degree of the variation for 
each object pivot points 601 is high" (Applicants, page 23, line 1). Therefore, "for a headwear 
600, such as hats or visor, the same problem exists. Usually caps have a uniform bottom line, but 
women's hats have various bottom lines" (Applicants, page 23, line 6-8). 
Novel Usage Of The Two-Level Automatic Face Detection Processes 
For The Verification Of Face In The Real-Time Face Tracking Window 
Is Clearly Foreign To Gao And Soatto 

Gao and Soatto are clearly foreign to the novel usage of the two-level automatic face detection 
processes for the verification purpose in applicants' system. Applicants used a first global face 
detection process for an initial face detection and a second local face detection process for the 
verification of face within the boundary of the face tracking window. Applicants' novel usage of 
the automatic face detection processes in the two levels for the verification purpose enables a 
novel and efficient real-time tracking for accurate face image. 

Applicants disclosed, "The verification 206 step uses one of the two types of face detection in 
the FET system. First face detection 204 is processed within entire frame in order to find whether 
a user exists in front of the system or not. Second local face detection 303 is processed within a 
small tracker window for verification 206 purpose. The verification 206 step has two goals. First 
it verifies whether the user is still using the system. Second, it also verifies whether the tracker is 
correctly tracking 205 the face and returns the correct face image 800 to the facial feature 
detection module 201. " in (Applicants, page 13, lines 1-8). 
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In applicants 5 system, the "verification 206 using the second local face detection 303 process is 
used in order to ensure that an accurate face image 800 is fed into the next module." (Applicants, 
page 8, lines 18-20), while Gao discussed about the facial shape detection process in (Gao, 
column 14, lines 8-29) as the last O.A. also noted. As Gao explicitly disclosed, the facial shape 
detection process in Gao is to assist "in making the correct shape determination" from a cosmetic 
standpoint and provide a "convenience to the user" for selecting eyewear (Gao, column 13, line 
66 - column 14, line 7). Determining a correct shape of face and providing a convenience to the 
user in Gao are clearly different from ensuring that an accurate face image is fed into the next 
module as a part of the real-time face tracking process in applicants' system. Therefore, Gao is 
clearly foreign to the novel usage of the face detection processes for the verification purpose in 
applicants 5 system. Soatto does not do this. 

Furthermore, applicants disclosed, "use of asynchronous multi-threading programming for the 
face detection 204 is another big factor to maximize the efficiency 55 (Applicants, page 15, lines 
3-4). Neither Gao nor Soatto do this. 

Motion-based Touch-Free Interaction Is Clearly Foreign To Gao And Soatto 

Motion-based touch-free interaction is clearly foreign to Gao and Soatto, and it is a novel 
approach over any prior art or any combination thereof. 

Regarding claim 5, as now applicable to claim 22, the last O.A. noted that "the combination of 
Gao and Soatto disclose that the method further comprises a step for processing touch-free 
interaction between said individual or said plurality of people and the system, whereby said 
touch-free interaction with said system enables said individual or said plurality of people to 
choose said prepared virtual object images on a means for displaying to superimpose onto said 
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facial image", referring to (Soatto column 10 line 15 to column 1 1 line 26). The O.A. noted also 
that "Soatto disclose tracking and the face and superposing the eyeglass frame throughout 
multiple frames of video in a touch free environment" in (Soatto column 10 line 15 to column 
11 line 26). 

In the referred paragraphs, Soatto explicitly disclosed methods in which a feature template is 
selected. Specifically, Soatto explicitly disclosed, "After the eyeglass frames image has been 
superimposed on the image of the person's face, a feature template is selected 212, either 
manually by having the user click on relevant features such as the corners of the eyes, nostrils, 
eyebrows etc. of the image, or automatically by performing 214 correlation with a stored 
database of templates for eyes, mouths, eyebrows, nostrils etc. The location on an image plane 
that scores the highest correlation with the template is chosen as the candidate position for the 
corresponding feature. Each feature is then tracked from image to image by performing 
correlation only in a neighborhood of the corresponding position at the previous image." (Soatto, 
column 10, lines 15-27). 

Whereas, applicants disclosed "The user can select different virtual objects by a motion-based 
touch-free user interaction/selection process 105." (Applicants, page 7, lines 12-13) and "The 
superimposed image 209 can be selected by the touch-free user interaction 105." (Applicants, 
page 11, lines 1-2). 

Soatto is clearly foreign to the idea of using a motion-based touch-free interaction for selecting 
different virtual objects before superimposing the selected virtual objects onto the user's face 
image. Soatto 5 s features are "the corners of the eyes, nostrils, eyebrows etc. of the image" 
(Soatto, column 10, lines 18-19), not "different virtual objects" (Applicants, page 7, line 12) or 
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"The superimposed image 209" (Applicants, page 1 1, line 1). Soatto's selection methods, even 
though the selected objects are not the same as the "The superimposed image 209" in 
(Applicants, page 1 1, line 1), are either "manually by having the user click on relevant features", 
which does not explicitly disclose "a motion-based touch-free user interaction/selection process 
105." (Applicants, page 7, lines 12-13), or "automatically by performing 214 correlation with a 
stored database of templates for eyes, mouths, eyebrows, nostrils etc.", which is clearly different 
from the selection by user's intended interaction without touching any device. The motion-based 
touch-free interaction is an interaction between the user and the system without touching any 
physical device, such as keyboard, touch-screen, or mouse, but interaction using a motion, such 
as hand gesture, in applicants' system. 

Gao mentioned about employing "speech recognition technology to accomplish interactive 
functions between the user and the system" for an alternative embodiment in (Gao, column 7, 
lines 4-7), but Gao's primary interface is a touch screen along with a keyboard and barcode 
scanner as Gao disclosed "Monitor 26 provides direct interactive capability to the user, allowing 
the user to navigate through the system and make selections by touching the screen. In this 
preferred embodiment, the touch screen is the primary interface mechanism between the user and 
the system. This feature contributes to the "user friendly" aspect of the system. An alternative 
embodiment may employ speech recognition technology to accomplish interactive functions 
between the user and the system. A keyboard 28 and barcode scanner 30 are also provide as 
input devices to the system." (Gao, column 6, line 66 - column 7, line 16). Therefore, Gao and 
Soatto, or any combination thereof is clearly foreign to the application of the motion-based 
touch-free interaction in the manner proposed in applicants' approach. 
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Novel Approach In The Incorporation Of Multiple Real-Time Facial Feature Detection 
Processing Algorithms And The Block Processing Is Clearly Foreign To Gao And Soatto 
Or Any Combination Thereof 

There are fundamental differences between applicants' system and Gao and Soatto or any 
combination thereof in processing the facial feature detection and application of the technology. 
Gao and Soatto did not disclose the novel approach in the fusion of multiple real-time facial 
feature detection processing algorithms. In addition, the novel usage of block processing in 
applicants' system is clearly foreign to Gao and Soatto or any combination thereof. 
In the pupil detection, Gao regarded "When a photograph is taken by system 10, the front of the 
user's face must appear in the center portion of the photograph." (Gao, column 11, lines 20-30). 
This assumption was based on the manual calibration procedure, in which Gao required a fixed 
distance between the video camera and the person who sits at the preset image acquiring location 
and Gao used manual adjustment by the operator for the two adjustable vertical bars (Gao, 
column 7, lines 34-57). Gao is clearly foreign to the concept of automatic tracking of the face 
and dynamic adjustment of the user movement and size change with regard to the automatic 
facial feature detection. In the following procedure for the pupil detection, Gao disclosed nose tip 
detection for centerline, black objects search in the binary image for the iris candidates, and 
circle-like edge search to fine tune the pupil center positions (Gao, column 11, line 20 - column 
13, line 14), but Gao is clearly foreign to the novel approach in the incorporation of multiple 
real-time facial feature detection processing algorithms, such as the incorporation of the color- 
based facial feature detection method and the mean-crossing method, or block processing as 
proposed in applicants' system. 
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In the preferred embodiment of Soatto, "the location of the center of the pupil of the eyes is 
determined in two steps. First, the location is coarsely determined by the user who indicates the 
approximate location utilizing a pointing device (the user is requested to click on the center of 
each eye). Second, the location is refined automatically using a template search" (Soatto, column 
7, lines 24-31). Although, Soatto used an algorithm for face localization to locate the position of 
the eyes in the image, in an alternative embodiment (Soatto, column 7, lines 46-50), Soatto 's 
semi-manual approach in the preferred embodiment or the usage of face localization in the 
alternative embodiment did not explicitly show the novel approach in the incorporation of 
multiple real-time facial feature detection processing algorithms, such as the incorporation of the 
color-based facial feature detection method and the mean-crossing method, or block processing 
as proposed in applicants' system. 

In (Soatto, column 10, lines 24-35), Soatto explicitly disclosed a feature template is selected 
either manually by a user or automatically by performing correlation with a stored database of 
templates, as in "a feature template is selected 212, either manually by having the user click on 
relevant features such as the corners of the eyes, nostrils, eyebrows etc. of the image, or 
automatically by performing 214 correlation with a stored database of templates for eyes, mouths, 
eyebrows, nostrils etc." Soatto is clearly foreign to the idea of incorporation of multiple real-time 
facial feature detection processing algorithms, such as the incorporation of the color-based facial 
feature detection method and the mean-crossing method. Applicants disclosed "In this exemplary 
embodiment shown in FIG. 2, the mean-crossing method may give better results overall, so it 
may be weighted more than the color-based method in the later fusion 405 step." (Applicants, 
page 10, lines 16-19). The "fusion 405 step" of the multiple facial feature detection algorithms 
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and the concept of one facial feature detection algorithm is "weighted more than" another facial 
feature detection algorithm is clearly foreign to Soatto. 

Applicants disclosed "For the facial feature detection 207, the FET system may incorporate 
multiple algorithms, for which real-time processing is possible. In the exemplary embodiment 
shown in FIG. 2, the color-based method may be used. The mean-crossing method suggested by 
C.H. Lin and J.L. Wu in "Automatic Facial Feature Extraction by Genetic Algorithms", IEEE 
transactions on image processing, Volume 8, no. 6, pages 834-845, June 1999, may be used, 
mainly due to their speed, simplicity, flexibility to the environmental change." (Applicants, page 
10, lines 10-16). 

In applicants' system, "The block processing 402, 404 looks at the neighboring feature candidate 
points and count into the final feature candidate position. The modified function is described as 
follows. 



^W(^o) = max 



E 2X('o) 



where b 0 is the evaluated block, F gJ (t 0 )is the cost of the candidate point at (ij) within the block, 

bx 0 , bx x , by 0 , and by x denote the starting and ending position of the block." (Applicants, page 

20, lines 5-12). The F(t 0 ) in (Applicants, page 19, lines 8-20), can be an exemplary F tJ (f^) 9 the 

cost of the candidate point at (ij) within the block. The block processing is to find the maximum 
cost among all the calculated costs for each pixel within a block, not to remove isolated small 
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objects within a window using morphological erosion. This is clearly foreign to Gao and Soatto 
or any combination thereof. 

Non-Automatic Initialization Step For The Facial Feature Detection And Cumbersome 
Calibration Step In Gao And Soatto Are Obstacles That The Applicants' System 
Overcomes 

Gao required a non-automatic initialization step for the facial feature detection and Soatto 
required a cumbersome calibration step, which the applicants' system considers as obstacles to 
overcome. 

In Gao, "Before system 10 can accurately superimpose a frame image onto a face image, and 
calculate relevant frame parameter data, system 10 must be initially calibrated. The key here is to 
obtain the actual physical scale for the face image." (Gao, column 7, lines 34-38). In the 
calibration procedure, Gao requested "two adjustable vertical bars are computer generated on 
touch screen 26, and are adjusted by the operator so that one is at the left temple and the other is 
at the right temple of the frame" (Gao, column 7, lines 45-48) with the fixed distance between 
the video camera and the sitting location of the person. 

In Soatto 's three-dimensional modality, "two or more cameras are rigidly mounted on a rig (a 
rigid infrastructure of pedestals), spaced about 1 m apart, and camera parameters are calibrated 
using standard techniques for stereo calibration as is well known." (Soatto, column 8, lines 19- 
24). 

Applicants disclosed "When a user appears in front of the FET system, the user's face is detected 
204 within the continuous input images 203 and the system is automatically initiated to begin the 
entire initialization process 200." (Applicants, page 8, line 21 - page 9, line 2). 
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Applicants also disclosed "The super- imposer receives facial feature detection information 501 
about the face measurement from the facial feature detection module 201, and it initialize its 
internal variables with the information, such as eye position 608, left face-line 612, right face- 
line 613, vertical center line 614 of the face image 800, uppermost-line 606 of the face image 
800, and lowermost-line 607 of the face image 800." (Applicants, page 22, lines 1-5). 
Applicants' system does not process 3 -dimensional models using two or more cameras as in 
Soatto, so applicants' system does not need any stereo calibration. In applicants' system, "The 
image-capturing devices automatically adjust the height of the viewing volume according to the 
height of the user." (Applicants, page 7, lines 13-14) when a pan/tilt enabled camera, such as 
"Sony EVI-D30" (Applicants, page 7, line 17-18) is used. Neither Soatto nor Gao do this. 
Novel Approaches In Real-Time Face Tracking, Facial Feature Detection, And 
Superimposition That Overcome The Issues And Challenges In Uncontrolled Environment 
Are Clearly Foreign To Gao And Soatto 

Real-time face tracking, facial feature detection, and superimposition that overcome the issues 
and challenges in the uncontrolled real-world environment are clearly foreign to Gao and Soatto. 
Gao disclosed removal of the background in details, especially in the section for "Frame Image 
Pre-Processing" (Gao, column 16, line 25 - column 19, line 57), such as the usage of "pure color 
paper" to remove the background in the "transparent" approach (Gao, column 17, lines 35-43). 
However, Gao did not explicitly disclose real-time face tracking, facial feature detection, and 
superimposition that overcome the issues and challenges in uncontrolled real-world environment, 
such as cluttered background or lighting variation. 
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In the three-dimensional modality of Soatto, "A digital picture of the person's face is then 
acquired simultaneously by the two (or more) cameras against a uniformly colored background 
(e.g. black) using controlled light (e.g. two diffusing lamps)." (Soatto, column 8, lines 41-44). 
Whereas, applicants disclosed "With cluttered background and other changing factors in the 
environment such as light variation or noises, the tracker might track false face region. In any of 
those cases, we need to verify whether the tracker is tracking 205 the user's face correctly or not. 
For this reason, the system is designed to run the local face detector 303 within the smaller face 
tracker window. 55 (Applicants, page 13, lines 11-15). 

Applicants also disclosed "This simple modification makes the approach tolerant to the 
environmental variation and noise in the cluttered background." (Applicants, page 20, lines 13- 
14). 

Automatic and Dynamic Change Handling For Facial Features Size And User Position In 
Applicants' System Is Clearly Foreign To Gao 

Gao assumed a fixed user position and a fixed size for the face and facial features, whereas 
applicants' approach handles the dynamic movement and changing size of the user position and 
facial features in real-time and automatically. 

Gao did not explicitly disclose how to track the user's face images in a continuous video frame 
in real-time and superimpose the virtual object images on to the user's face images continuously 
in real-time. In (Gao, column 7, lines 34-57), Gao required a fixed distance D between the user 
and the person. Gao also required the person to sit "at the pre-set image acquiring location" in 
the vicinity of the system. Gao also assumed that the physical scale is fixed in "all subsequent 
physical size calculations performed in system 10 will be based on scale s" (Gao, column 7, lines 
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54-56), by using the "(3) two adjustable vertical bars", which "are computer generated on touch 
screen 26, and are adjusted by the operator so that one is at the left temple and the other is at the 
right temple of the frame (i.e., the front of the frame is closely situated between the two vertical 
bars), and then processing unit 36 calculates the distance d between those two bars internally, 
and the frame size parameters, EYE and DBL, are entered from keyboard 28 by the operator;" 
When the user's head orientation is changed or slanted, the "(3) two adjustable vertical bars" 
have to be re-adjusted to reflect the change in the scale. However, Gao did not disclose how to 
handle the cases. 

Gao also used fixed sub-images with constant numbers, "In order to find the nosetip, a 100X80 
pixel sub-image is selected for processing. The sub-image is extracted from the original image, 
80 pixels below eye level and 50 pixels to left and right of the image center." and "Next, an area 
s(x 5 y) is selected with a width 60 pixels around the eye level." (Gao, column 11, lines 46-61), 
which clearly shows Gao assumes a fixed size of the face image, and Gao are not be able to 
handle the changing size of the user's face. 

With regard to Soatto, there is a difference between Soatto's correlation search for the "Two- 
and-a-Half-Dimensional Model Visualization" in (Soatto, column 10, line 7 - column 11, line 
37) and the automatic and dynamic movement handling for facial features and user position in 
applicants' system. 

Soatto disclosed "the system requests scale information, such as inter-ocular distance." (Soatto, 
column 5, line 44) to the user for manual input, and "the user changes the shape by moving 
sliders 128-146 that control the horizontal and vertical position of each control point." (Soatto, 
column 6, lines 3-4). 
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Soatto also disclosed "a feature template is selected 212, either manually by having the user click 
on relevant features such as the corners of the eyes, nostrils, eyebrows etc. of the image, or 
automatically by performing 214 correlation with a stored database of templates for eyes, mouths, 
eyebrows, nostrils etc." (Soatto, column 10, lines 16-21), then "Each feature is then tracked from 
image to image by performing correlation only in a neighborhood of the corresponding position 
at the previous image" (Soatto, column 10, lines 24-26). 

However, Soatto clearly and explicitly disclosed that this process is performed "After the 
eyeglass frames image has been superimposed on the image of the person's face," (Soatto, 
column 10, lines 15-16), not before the eyeglass frame image is superimposed on the image of 
the person's face as it is done in applicants' system. Soatto 's "stored database of templates" is 
clearly different from "rectangular facial component boxes" (Applicants, page 10, line 2) in 
applicants' system, which dynamically change depending on the rough face region measure 
based on the "user's face image 800 position and size" by the face-tracking 205 process, using 
skin-color, Viterbi-algorithm based method, or any real-time face-tracking algorithm, and 
verification process (Applicants, page 9, lines 10-15 and page 12, lines 9-21). 
Applicants' system does not require the distance between the user and the system to be fixed. 
Applicants' s system does not require the user to sit in a fixed position either. Applicant's system 
does not use fixed vertical bars or any fixed templates at all. 

Applicants' system adapts to changing position and size of the person based on the face tracking 
process, as disclosed in, "After the initial global face detection 204, the face-tracking 205 
process provides the information about the user's face image 800 position and size." (Applicants, 
page 9, lines 10-12). In applicants' system, the users can move around in the vicinity of the 
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system, and "The FET system is adaptable to the change of the face image 800 size in the 
continuous video image sequences." (Applicants, page 1 1, lines 11-12). Applicants also 
explicitly disclosed, "Usage of a fixed size template is strongly prohibited in the FET system" as 
said in, "The reduced region of interest may serve as the initial template for each facial features. 
However, it has to be emphasized that these are not fixed templates. These rectangular facial 
component boxes dynamically change depending on the rough face region measure. Usage of a 
fixed size template is strongly prohibited in the FET system." (Applicants, page 9, line 22 - page 
10, line 4). 

Handling Low-Resolution Images For The Facial Feature Detection And Superimposition 
Of The Virtual Object Images In Applicants' System Are Foreign To Gao And Soatto 

High resolution images were assumed in Gao, and handling low-resolution images for the facial 
feature detection and superimposition of the virtual object images are foreign to Gao. 
Gao disclosed "The second subsystem interfaces with the user to acquire a high resolution, full 
color digital photograph of the user's face, and to digitally superimpose a frame image from the 
database on the image of the user's face." (Gao, column 6, lines 31-35), and Gao is foreign to the 
concept of real-time and automatic facial feature detection in low-resolution face images. 
Regarding Soatto, "Alternatively, the correlation search is performed in a coarse-to-fine fashion 
by allowing a template to exist at several levels of resolution, and using feature locations at 
coarse resolution to initialize the search at the finer resolution." (Soatto, column 10, lines 29-33) 
was disclosed. Although Soatto used multiple resolution templates, Soatto 's approach is still 
limited by the number of templates, more explicitly "several levels of resolution" (Soatto, 
column 10, line 31) in "a stored database of templates" (Soatto, column 10, line 20). If the size of 
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facial features is beyond the limitation of the "several levels of resolution" (Soatto, column 10, 
line 31) in "a stored database of templates" (Soatto, column 10, line 20), Soatto' s approach may 
not be applicable. 

Applicants' system handles the low-resolution images as disclosed "it is a necessary that the 
algorithm in the system has to work with the low-resolution images" (Applicants, page 17, lines 
18-19) as a part of the discussion for the challenges in real-time and automatic facial feature 
detection process (Applicants, page 15, line 19 - page 17, line 19). Applicant's system does not 
have limitation for the number of levels in the resolution. Applicants' system can handle the low- 
resolution images as long as "the face-tracking 205 process provides the information about the 
user's face image 800 position and size." (Applicants, page 9, lines 10-12). 
Novel And Unobvious Approaches In Applicants' System Are Clearly Foreign To Gao And 
Soatto Or Any Combination Thereof And Should Not Be Underestimated Or 
Misunderstood. 

Novel and unobvious approaches in applicants' system are clearly foreign to Gao and Soatto as 
discussed above for the eight differences in the fundamental approaches. The significant 
differences in the approaches between applicants' system and Gao and Soatto or any 
combination thereof should not be underestimated or misunderstood. 

Since The Fundamental Approaches In Applicants 9 System Clearly Foreign To Gao And 
Soatto, Gao And Soatto Do Not Contain Any Justification To Support Their Combination 
And Even If Gao And Soatto Were To Be Combined, The Proposed Combination Would 
Not Show All The Novel Features Of Claim 22 
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There is no justification, in Gao and Soatto, or in any other prior art separate from applicants' 
disclosure, which suggests that these references be combined, much less be combined in the 
manner proposed. 

For example, Gao assumed a fixed user position and a fixed size for the face and facial features. 
Soatto disclosed "stored database of templates" for correlation search from image to image. As 
discussed above, although Soatto 's "stored database of templates" is clearly different from 
"rectangular facial component boxes" (Applicants, page 10, line 2) in applicants' system, Soatto 
used a set of variable templates. Therefore, there is no justification to support the combination of 
their contradictory approaches to produce the same results by the "rectangular facial component 
boxes" (Applicants, page 10, line 2) in applicants' system, which dynamically change depending 
on the rough face region measure based on the "user's face image 800 position and size" by the 
face-tracking 205 process, using skin-color, Viterbi-algorithm based method, or any real-time 
face-tracking algorithm, and verification process (Applicants, page 9, lines 10-15 and page 12, 
lines 9-21). 

Likewise, Gao's high resolution image assumption could be contradictory to Soatto' s usage of 
"several levels of resolution" (Soatto, column 10, line 31) in "a stored database of templates" 
(Soatto, column 10, line 20). Some of the "several levels of resolution" (Soatto, column 10, line 
31) may be intended for low-resolution images although the number of templates is limited as 
several levels of resolution. Therefore, there is no justification to support the combination of 
their contradictory approaches to produce the same results as applicants' system can produce. As 
discussed above, applicants' system can handle the low-resolution images as long as "the face- 
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tracking 205 process provides the information about the user's face image 800 position and 
size." (Applicants, page 9, lines 10-12). 

Since the fundamental approaches in applicants' system are clearly foreign to Gao and Soatto, 
even if Gao and Soatto were to be combined in the manner proposed in the last O.A., the 
proposed combination would not show all the novel features of claim 22. 
For example, even if Gao and Soatto were to be combined, they would not show the non- 
eyewear facial image enhancement system, novel usage of the two-level automatic face detection 
processes for the verification of face in the real-time face tracking window, motion-based touch- 
free interaction, incorporation of multiple real-time facial feature detection processing algorithms 
and the block processing, automatic initialization, and real-time face tracking, facial feature 
detection, and superimposition that overcome the issues and challenges in uncontrolled 
environment, since neither Gao nor Soatto do any of these. 
The Novel Features Of Claim 22 Produce New And 
Unexpected Results And Hence Are Unobvious And 
Patentable Over These References Under § 103 

The block processing in the application of the plurality of facial feature detection algorithms, 
such as the mean-crossing algorithms, and the integration design of each key process in 
applicants' system for real-time and automatic facial image enhancement in uncontrolled 
background environment as proposed in applicants' disclosure produce new and unexpected 
results and hence are unobvious and patentable over these references. 
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The Dependent Claims Are a Fortiori Patentable Over Gao And Soatto 

New dependent claims 23 to 32 incorporate all the subject matter of claim 22 and add additional 
subject matter, which makes them a fortiori and independently patentable over the reference. 
Claim 23 further adds a step of repeating the steps from (a) to (g) in claim 22 for a plurality of 
persons. Using multi-threaded face detection process, applicants' system can process multiple 
persons' facial image enhancement. Multiple persons processing is possible in applicants' system 
since it is not required for the positions of the persons are fixed, and as stated above "multi- 
threading programming for the face detection 204" (Applicants, page 15, line 3) is used in 
applicants' system. Neither Gao nor Soatto do this in the manner proposed in applicants' system. 
Claim 24 further adds "a step of estimating regions of interest for each facial feature in said face 
dynamically, whereby said regions of interest change according to the results from the step of 
verifying said face within the boundary of the face tracking window, and whereby said regions of 
interest are used as boundaries for detecting each facial feature, such as eyes, nose, and mouth on 
said face." As discussed above, "The reduced region of interest may serve as the initial template 
for each facial features. However, it has to be emphasized that these are not fixed templates. 
These rectangular facial component boxes dynamically change depending on the rough face 
region measure. Usage of a fixed size template is strongly prohibited in the FET system." 
(Applicants, page 9, line 22 - page 10, line 4). Neither Gao nor Soatto do this in the manner 
proposed in applicants' system. 

Claim 25 further adds "a step of applying fusion algorithms and geometrical constraints to said 
facial feature coordinate information." Neither Gao nor Soatto do this in the manner proposed in 
applicants' system. 



Appn. Number 10/609,245 (Namsoon Jung, et aU GAU 2624 Amnt. B contd. 32 of 37 



Claim 26 further adds "a step of smoothing said facial feature coordinate information." Neither 
Gao nor Soatto do this in the manner proposed in applicants' system. 

Claim 27 further adds "a step of storing a history of a plurality of said facial feature coordinate 
information and applying a smoothing algorithm for the current facial feature coordinate using 
said history of said facial feature coordinate information." Neither Gao nor Soatto do this in the 
manner proposed in applicants' system. Gao disclosed, "having the standard facial shapes stored 
in memory, as a reference" (Gao. column 14, lines 14-15), and Soatto disclosed, "reference 
frame" (Soatto, column 10, lines 15-54). However, neither Gao nor Soatto disclosed "storing a 
history of a plurality of said facial feature coordinate information" and "applying a smoothing 
algorithm for the current facial feature coordinate", using the history of the facial feature 
coordinate information. The "constellation of positions of feature locations is interpreted as 
moving according to a Euclidean planar motion" in (Soatto, column 10, lines 41-43) did not 
disclose temporal attributes of the stored "history of a plurality of said facial feature coordinate 
information" in applicants' system either. 

Claim 28 further adds "a step of combining the face detection process and the real-time face 
tracking process". Neither Gao nor Soatto do this in the manner proposed in applicants' system, 
since the approaches in using the face detection process and real-time face tracking in the 
manner proposed in applicants' system are foreign to Gao and Soatto. 

Claim 29 further adds "a step of applying said block-processing to each of said plurality of facial 
feature detection approaches." The definition of block-processing as discussed above and in the 
applicants' specification is clearly foreign to Gao and Soatto or any combination thereof. 
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Therefore, application of the "block-processing to each of said plurality of facial feature 
detection approaches" is further foreign to Gao and Soatto or any combination thereof. 
Claim 30 recites the step of superimposing said virtual object images onto said facial images 
automatically and dynamically in real-time in claim 22, further comprises steps of: 
(a) preparing virtual object images, (b) validating said facial feature coordinate information, 
(c) smoothing, (d) aesthetic processing, and (e) processing final superimposition. Neither Gao 
nor Soatto do these steps in the context of superimposing said virtual object images, including 
non-eyewear, onto said facial images, automatically and dynamically in real-time as discussed 
above with respect to claim 22. 

Claim 3 1 further adds "a step for setting pivot points in said virtual object images" to claim 30. 
Neither Gao nor Soatto do these steps in the context of superimposing said virtual object images, 
including non-eyewear, onto said facial images, automatically and dynamically in real-time as 
discussed above with respect to claims 22 and 30. 

Claim 32 further adds "a step for processing rotation and translation of said virtual object 
images" to claim 30. Neither Gao nor Soatto do these steps in the context of superimposing said 
virtual object images, including non-eyewear, onto said facial images, automatically and 
dynamically in real-time as discussed above with respect to claims 22 and 30. 
Accordingly applicants submit that the dependent claims are a fortiori patentable and should also 
be allowed. 
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The Rejection Of Claim 13 on Gao and Soatto Overcome 

Regarding claim 13, the last O.A. noted, "please see the rejection of claims 1 and 2 and see 
figure 2 of Soatto which discloses an apparatus". 

Claim 13 has been canceled and the limitations in claim 13 have been integrated into the newly 
written claim 33 as limitations of the newly made independent claim to define patentably over 
the reference. 

Applicants request reconsideration of the rejection, as now applicable to claim 33, for the same 
reasons as discussed above with regard to claim 22. 

The Dependent Claims Are a Fortiori Patentable Over Gao And Soatto 

New dependent claims 34 to 43 incorporate all the subject matter of claim 33 and add additional 
subject matter, which makes them a fortiori and independently patentable over the reference. 
Regarding claims 14, 15, and 16, they have been canceled. 

Regarding claim 17, it has been canceled and all of the limitations in claim 17 have been 
integrated into the newly written claim 33 as limitations of the newly made independent claim to 
define patentably over the reference. 

Claims 18, 19, 20, and 21 have been rewritten as new claims 35, 36, 42, and 41, respectively, to 
more particularly define the invention in a patentable manner over the cited prior art. Applicants 
request reconsideration of the rejection regarding claims 18, 19, 20, and 21, as now applicable to 
new claims 35, 36, 42, and 41, respectively, for the same reasons as discussed above with regard 
to claims 24, 25, 3 1, and 30. 

Regarding new dependent claims 34, 37, 38, 39, 40, and 43, they encompass substantially the 
same scope of the invention as that of claims 23, 26, 27, 28, 29, and 32, respectively, in addition 
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to an apparatus and means for performing the method steps of claims 23, 26, 27, 28, 29, and 32, 
respectively. Therefore, please refer to claims 23, 26, 27, 28, 29, and 32 for the reasons of 
patentability with respect to the new dependent claims 34, 37, 38, 39, 40, and 43. 
Accordingly applicants submit that the dependent claims are a fortiori patentable and should also 
be allowed. 
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CONCLUSION 

For all the above reasons, applicants submit that the specification and claims are now in proper 
form, and that the claims all define patentably over the prior art. Therefore they submit that this 
application is in condition for allowance now, which action they respectfully solicit. 

Conditional Request for Constructive Assistance 

Applicants have amended the specification and claims of this application so that they are proper, 
definite, and define novel structure, which is also unobvious. If, for any reason this application is 
not believed to be in full condition for allowance, applicants very respectfully request the 
constructive assistance and suggestions of the Examiner pursuant to M.P.E.P. § 2173.02 and § 
707.07(j) in order that the undersigned can place this application in allowable condition. 

Very respectfully, 





Namsoon Jung 



Rajeev Sharma 



Applicants Pro Se 



403 South Allen Street, Suite 101 



State College, PA 16801 



Tel. (814) 867-8977; Fax (814) 867-8957 
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